When most people think of music, they typically think of experiencing it only in an auditory manner. Of course, the long history of musical accompaniment to visual media such as movies, television, and computer games demonstrates that people enjoy experiencing music in conjunction with a visual presentation. In the realm of digital music, computer music visualization systems exist for generating animated imagery in real-time for a piece of music. Existing music visualization systems are typically configured with preset animation parameters that are adjusted in real-time in response to changes in basic properties of an input piece of music such as the music's loudness and frequency spectrum
Traditionally, commercial music has been released in stereo format. Typically, music producers and artists write, record, or synthesize different layers of audio on a multitrack hardware or software system. Once the composition is finished, a mixed mono or stereo track is exported, where all instruments and elements are now mixed at the signal level. Optionally, the mixed mono or stereo track can go through a mastering process, which will also output a processed mono or stereo mix.
Although the stereo format has been massively adopted for listening purposes, it has certain limitations. One limitation is that once layers/sources have been mixed, it is generally not possible to listen to an isolated element from the original composition because the source audio elements overlap in the mixed time domain signal and often in the corresponding frequency domain as well. Audio equalizers have traditionally been used to emphasize or attenuate certain source elements in the underlying mix, but with limitations in terms of the audible results.
Basic visualization of certain features from the different elements or sources in a music mix have been developed over the years. For example, one can still achieve some degree of discrimination and create visualizations of the instantaneous energy present at different frequency bands within a mix. Since certain musical instruments concentrate most of their energy in specific regions within the frequency spectrum, creation of basic visualizations that appear to respond to individual elements or sources within the mix are possible. An example of a basic visualization is an EQ visualization as depicted in FIG. 9. As shown, colors in the EQ visualization are assigned to different frequency bands and the instantaneous energy computed at each frequency band is represented by the height of some vertical bars. However, the source element discriminations involved in basic visualizations typically are not very precise and typically do not involve precise identification of the underlying sources in the input audio mix to create the visuals. With current visualization techniques, it is generally not possible to create precise visualization using features that corresponding to individual audio sources within a digital music input mix.
The approaches described in this section are approached that could be pursued, but not necessarily approached that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approached described in this section qualify as prior art merely by their inclusion in this section.